The HDR Photographic Survey

High-dynamic-range (HDR) imaging is one of the remaining frontiers for significant advancement in consumer and professional color imaging systems. Systems for the capture, processing, and display of HDR images are gradually moving from the research labs and specialized applications to more mainstream usage and impact on consumer photography. The HDR Photographic Survey is a research project and public-domain database of images and scene data aimed at improving these systems by providing images and data that can be freely used by researchers around the world. This allows for more efficient testing and improvement of HDR algorithms and displays through enhanced inter-comparisons of results from various researchers and the availability of images with colorimetric and color appearance reference data from the original scenes. This paper describes the process of collecting the images and scene data, the system characterization, and the creation and use of the database. Introduction In the 1860’s and 1870’s outstanding and dedicated photographers such as Timothy O’Sullivan, William Henry Jackson, John K. Hillers, and others took part in expeditions to the American west carried out by the U.S. Geological Survey to document the natural wonders that were being discovered.[1] These photographic surveys of the American west, as well as similar surveys around the world, produced immensely popular and politically important images. For example, the works of William Henry Jackson have been credited with being largely responsible for the creation of the US National Park Service and the establishment Yellowstone National Park. As described by Keiley,[2] “W.H. Jackson, who continues to this day to serve as a collaborator on national park studies, was a member of the Hayden party. He obtained a remarkably fine series of Yellowstone photographs, samples of which Dr. Hayden placed on the desks of all Senators and Congressmen ... influencing the passage of the National Park Act.” The profound influence of imaging on political matters had begun. The magnitude of these accomplishments is only amplified when one contemplates the imaging technology of the time. O’Sullivan, Jackson, and their contemporaries used large-format (up to 20x24 inches) cameras, glass plates, and the wet collodion process that required the plates to be coated with a mixture of collodion and potassium iodide, immersed in a sensitizing solution of silver nitrate, then exposed, processed and fixed before the plates dried; all in the field. These pioneers, of course, were followed by the likes of Ansel Adams, whose images furthered the cause of preservation and documented the US National Parks in fine detail.[3] Ansel Adams can also be considered a technical innovator in imaging science and an early master of HDR imaging. He was also well-known for teaching and sharing his knowledge.[4] His techniques, described as the Zone System, involved pre-visualization of the desired print while at the original scene, careful exposure of the photographic negative as a relatively-HDR record of the scene, and then careful printing with dodging, burning, and retouching to create lowdynamic-range prints that matched the actual or imagined appearance of the original scene. So it could be said that he recorded the appearance of the scene with local adaptation and then rendered that appearance in the darkroom. This is exactly what current researchers try to automate in some way with visual models of spatial adaptation aimed at developing HDR rendering algorithms.[5] In fact, Adams’ techniques have rather directly inspired one very successful HDR rendering algorithm.[6] Figure 1. Eighteen individual exposures, each separated by one stop, used to construct the Luxo Double Checker HDR characterization image. These photographic surveys serve as important documentation of our natural world as the global climate changes and have inspired more recent photographers to capture images of many of the same scenes over a century later. These efforts included the Rephotographic Survey Project in the 1970’s and the followup Third View at the turn of the this century.[7] The changes illustrated over the first 100 years, and then the more recent 30, are remarkable. These works point out the value of carefully recording the time, place, and imaging variables for the recorded scenes and provided inspiration for much of the current project, including the name. While photographic negatives have long been capable of capturing a wide dynamic range of scene information, and HDR-capable film, called XR for extended range, was created with multiple fast and slow emulsions to photograph atomic and nuclear bomb explosions,[8] the field has become of more widespread technical and scientific interest with the development of digital techniques of image rendering and multiple-exposure image capture. The recent, and comprehensive, book on HDR Imaging[9] illustrates the arrival of such techniques into the mainstream of imaging. Of particular note was the detailed exposition of techniques for creating HDR images from multiple photographic exposures by Debevec and Malik in the late 1990’s [10] which is the basis of the methods used to create the images described herein. Careful measurement of scene data for use in tone reproduction research also has a long history that is highlighted by the classic work of Jones and Condit [11] which also describes earlier work and highlights the importance of flare in both the imaging process and photometric measurements of scene elements. Research in HDR imaging has been very active in the development and psychophysical testing of computational algorithms to render HDR image data for display on low-dynamic-range displays or printers. Akyüz and Reinhard [5] and Kuang et al.[12] provide reviews of recent work. The development of HDR-capable displays has also grown recently with at least one such display [13] commercially available and methods being proposed for their use to enhance color imaging capabilities.[14] They are also being used in psychophysical experiments aimed at testing HDR imaging algorithms. Standard image data sets, such as the SCID images for example,[15] have been useful in imaging research to allow direct comparison of algorithm output, systems performance, and psychophysical results across research from various scientists and engineers worldwide. One aim of the HDR Photographic Survey is to provide such images in the public domain to researchers working on HDR systems and perception with a key feature being the inclusion of camera characterization data to allow conversion to accurate device-independent image data, colorimetric measurements of original scene elements, color appearance scaling of scene elements, and other scene data allowing increased utility of the images. Such a database of common images should allow improved evaluation of algorithms and displays along with a further understanding of the perception and visual modeling of HDR scenes and reproductions (e.g., improvements of models such as iCAM[16]). All of these works have inspired the current HDR Photographic Survey described in this paper with the hope that it will both document fully a variety of scenes in the early 21st century and provide scientific images and data to improve capture, processing, and display technologies for future generations of imaging systems and their users. Imaging Procedures Gone are the days where large-format, wet-collodion plates are required to make high-quality renditions of scenes. The HDR Survey images are based on the combination of multiple exposures made with a modern digital SLR. A Nikon D2x was used with a selection of lenses. Most of the images were obtained with a Nikon 17-55mm f/2.8 ED-IF AF-S DX Zoom-Nikkor lens. This is a very high-quality lens with minimal flare and an equivalent focal-length range of 25.5mm-85.5mm for 35mm format. In conditions of extreme potential flare, the brightest scene element was placed in the center of the frame to assure that most of the flare would also fall in the bright scene areas. The D2x is a professional digital SLR with a 12.4 megapixel CMOS sensor. Three reasons it was selected for this project were the availability of lenses, the high-speed performance (5 frames/sec.) and its auto-bracketing function that allows for nine exposures to be made with one stop increments in exposure time at a fixed aperture with one depression of the shutter release. This, combined with the speed, allows nine-exposure HDR sequences covering a nine-stop exposure range to be made in less than two seconds with sufficient light, a feature that is helpful for subjects that might tend to move. Figure 2. Linear (left) and local-adaptation (right) renderings of the HDR Luxo Double Checker image. Pertinent imaging information such as focal length, exposure times, number of exposures, lens used, etc. were recorded both manually and in the image meta-data for each scene. Typically, nine exposures in one-stop increments were captured for each scene. However some were captured with two sequences (18-stop range) or with three filter conditions to allow spectral (9-channel analysis). All images were saved in Nikon’s raw electronic format (NEF). They were converted to 16-bit TIFF files using the absolute minimal processing possible in Adobe Photoshop CS2 raw conversion utility. The TIFF files were then combined into a single HDR image using the Adobe Photoshop CS2 “merge to HDR” process. Resulting images were cropped for aesthetic purposes, retained in the native maximum resolution, and saved as OpenEXR files. The OpenEXR format retains the image data as 32-bit floating-point values.[17] These minimally-processed, and photometrically linear, OpenEXR files are what is available in the database. The process of white-balance is the main variable step in this procedure. Since may users would not desire true raw or absolute colorimetric images, it was decided that they should be white balanced. This was accomplished for each scene in the NEF-toTIFF conversion using the in-camera determined white point. The importance of this decision is discussed below in relation to camera characterization. Scene Measurement Procedures In addition to the images themselves, a range of data were collected about the original scenes. A Magellan eXplorist 600 GPS system with electronic compass was used to keep the photographer from getting lost and record the latitude, longitude, elevation, viewing direction, date, time, temperature, and weather conditions for each scene. These data allow imaging to be repeated at future times and those interested in using the images in simulation or rendering systems to accurately position natural lighting relative to the camera. In addition, for some scenes, images of a mirrored ball were also collected, however not in HDR. These allow modeling of the scene by defining the illumination from every direction simultaneously in a single exposure. For approximately half the captured scenes, colorimetric and color appearance data were also collected from the original scene. This was impossible in some cases because the scene content or lighting changed too quickly. In those cases, the absolute luminance of at least one scene element was measured to allow for some calibration of the image. A number of relatively uniform areas in the scene were identified and recored. A KonicaMinolta CS-100 spot colorimeter, capable of measuring 1° areas, was used to record absolute luminance (cd/m2) and CIE 1931 chromaticity coordinates. No special procedures were udnertaken to minimize flare in the colorimetric measurements since it is extremely scene dependen and was determined to be minimal in the calibration scene. The same spots were scaled for color appearance by the author using a previously developed and tested technique.[18] Lightness was scaled from 0 for perfect black to 10 for a perfect diffuse white. Chroma was scaled using the same perceptual magnitudes with the neutral at 0 and, for example, a chroma of 5 representing a stimulus that appears as different from neutral as a perfect black varies in lightness from a middle gray. Hue was scaled using NCS-type designations of percentages of the unique hues. Lastly, brightness was scaled for some scene elements using a magnitude estimation technique with several reference points (e.g. White under Full Moon = 3, White in Office Lighting = 50, White under Dull Sky = 100, Full Moon = 100, White under Bright Sun = 300, 60W Light Bulb = 400, Saturated Source = 1000, Sun = 10000) that were derived from the work of Stevens and Stevens [19] and Hunt.[20] In addition, summary statistics from the final images were collected and reported. These include the maximum, minimum, and mean luminance levels (sampled from significant image areas, not stray light or dark pixels), the dynamic range computed from these sample points, and the luminance multiplier, or luminance factor. This multiplier is the factor required to convert the relative data in the OpenEXR file into absolute luminance. This is necessary since no step in the process of creating the HDR image requires or retains the absolute luminance information and the resulting images tend to have a diffuse white with values of approximately 1.0 and maxima on the order of 10.0 with sigificant variation. Since some HDR rendering algorithms, and anyone trying to model and understand human perception, require absolute colorimetric information, these factors are the critical pieces of information that are missing from most available HDR images. Flare factors as described by Jones and Condit[11] are not reported for each scene since they are highly dependent on the scene configuration and elements selected for measurement. Camera Characterization The D2x camera system was colorimetrically characterized using two techniques. One was to carefully measure the spectral responsivity of the system using a calibrated monochromator system. The spectral sensitivities, thus obtained, can then be used to determine an approximate linear transformation between camera RGB values and CIE XYZ values. This technique will not be described in more detail here since a more direct technique based on HDR image measurements should be more directly related to the HDR images in the database. The two methods produced results in substantial agreement. An HDR scene was constructed in the laboratory to provide a significant HDR imaging challenge and provide a controlled scene that could be carefully measured to create a camera characterization matrix. The scene, called Luxo Double Checker, consists of a single light source with a clear incandescent bulb visible to the camera. This source, mounted in a Luxo desk lamp, directly illuminates a GretagMacbeth ColorChecker Chart. In the other half of the scene, separated by a black divider from the illumination, is a second ColorChecker Chart and a few other objects. These objects receive only indirect illumination reflected through the room. The dynamic range of the scene is approximately one-million-to-one when comparing the exposed lamp filament to the black areas on the left side of the scene (approximately 23 stops, or bits, of scene information). Even when imaged, the resulting HDR image retains a dynamic range of about 800,000:1 (19-20 stops, or bits, of real image information). The scene was imaged with 18 exposures in one-stop increments. Figure 1 is a rendering of all 18 individual exposures to illustrate the dynamic range of the scene and the need for multiple exposures to characterize it. Figure 2 includes two renderings of the final HDR image, linear and using an interactive local adaptation algorithm. Flare is always a limiting factor in image dynamic range. Jones and Condit [11] reported flare factors for their scenes that were a ratio of the measured luminance ratio to the photographic luminance ratio. They reported average flare factors of 2.5 for their 5x7 camera with a simple lens to minimize flare and 4.0 for typical cameras of the time. This 4.0 flare factor means that a scene with a 100:1 dynamic range would be reproduced with a 100:4 (or 25:1) dynamic range in the camera. It should be noted that their measurements are all for low-dynamic-range backlit scenes with no visible sources. Comparable measurements in the Luxo Double Checker scene using the black patch of the dim ColorChecker produce a flare factor of 1.4. While this seems only slightly better than the historical measurements made over 60 years ago, the flare factor is a relative metric that does not tell the whole story. In the Jones and Condit LDR scenes, the typical absolute levels of flare were about 6% of the scene maximum (about 2.4% for the photometer, giving the flare factor of 2.5). In the Luxo Double Checker scene, the absolute flare level was approximately 0.0001% in the image data and 0.00007% in the photometric measurements. While this is very scene dependent, measurement location dependent, and image composition dependent, it is clear that the modern imaging and measurement systems are far superior to the historical systems of Jones and Condit.[11] If this were not the case, HDR imaging simply would not be possible. Figure 3. Locations and numerical denotations of the scene elements measured and scaled in the Luxo Double Checker scene. A colorimetric characterization matrix was derived through linear regression based on measurements of the bright ColorChecker. Linearity and the low reported flare values were confirmed with the dim ColorChecker, but the CS-100 signal was too low to use for the model fit. The locations measured, and also scaled for appearance, in the scene are illustrated and denoted in Fig. 3.The model fits, with no intercept, were very good with R2 values of (0.998, 0.997, 0.992) for X, Y, and Z respectively. This linearity is expected since the process of assembling the HDR image creates photometrically linear image data. Thus no, characterization of the camera transfer function is necessary. Equation 1 is the resulting characterization matrix normalized such that equal RGB values produce D65 chromaticities. In some applications, other normalizations might be